Hopper: Decentralized Speculation-aware Cluster Scheduling at Scale – Public Review
نویسنده
چکیده
The huge volume of data available today has led to interest in parallel processing on commodity clusters. Data analytics distributed frameworks such as Hadoop, Spark, or Pregel are designed for parallel processing of a large amount of data. These frameworks break a computation job into small tasks that run in parallel on multiple machines, and aim to scale to very large clusters of inexpensive commodity computers. However, as jobs increase in size and complexity, scheduling these jobs so as to provide scalable and predictable performance becomes challenging.
منابع مشابه
Speculation and Decentralized Scheduling
As clusters continue to grow in size and complexity, providing scalable and predictable performance is an increasingly important challenge. A crucial roadblock to achieving predictable performance is stragglers, i.e., tasks that take significantly longer than expected to run. At this point, speculative execution has been widely adopted to mitigate the impact of stragglers. However, speculation ...
متن کاملSpeculation-aware Resource Allocation for Cluster Schedulers
Resource allocation and straggler mitigation (via “speculative” copies) are two key building blocks for analytics frameworks. Today, the two solutions are largely decoupled from each other, losing the opportunities of joint optimization. Resource allocation across jobs assumes that each job runs a fixed set of tasks, ignoring their need to dynamically run speculative copies for stragglers. Cons...
متن کاملEfficient Scheduling of Workflow in Cloud Enviornment Using Billing Model Aware Task Clustering
Cloud computing is a cost effective alternative for the scientific community to deploy large scale workflow applications.For executing large scale scientific workflow applications in a distributed hetereogenous enviornment ,scheduling of workflow tasks with the dynamic resources is a challenging issue.Moreover in a utility based computing like cloud which supports pay per use model of the resou...
متن کاملAn Effective Approach to Job Scheduling in Decentralized Grid Environment
Scheduling of jobs and resource management are the important challenging work in a grid environment. Processing time minimization of the jobs arriving at any computer site in a grid system is one of the major objectives in the research area of computing. In this paper, we propose a decentralized grid system model as a collection of clusters. We then introduce a decentralized job scheduling algo...
متن کاملFARMS: Efficient mapreduce speculation for failure recovery in short jobs
With the ever-increasing size of software and hardware components and the complexity of configurations, large-scale analytics systems face the challenge of frequent transient faults and permanent failures. As an indispensable part of big data analytics, MapReduce is equipped with a speculation mechanism to cope with run-time stragglers and failures. However, we reveal that the existing speculat...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2015